Long term data protection system and method

ABSTRACT

A file to be written can be partitioned into one or more partitions. Each such input partition is identified with a hash code and a group ID. Replica(s) of an input partition can be created to ensure a certain number of identical partitions. When a file is accessed, each partition that is read out can be checked for corruption. Corrupted readout partitions are then replaced with a validated replacement partition, whether the replacement partition is from a replica or from another file.

BACKGROUND OF THE INVENTION

The present invention is generally related to storage systems and inparticular to a system and method for providing reliable long termretention of data.

Events in recent times have identified the need for long-term storage ofdata. Businesses and data users in general have a need for archivingdata for long periods of time. Enterprises are interested in long termdata preservation, motivated in large part by government imposedregulations. For example, the U.S. Securities and Exchange Commission(SEC) requires exchange members, brokers, and dealers to preserverecords of trading accounts until the end of the account plus 6 years,and records of all communications, such as email with their customers,must be kept for a period of not less than 6 years under the SecuritiesExchange Act of 1934 Rule 17a-4. The National Association of SecuritiesDealers Inc. (NASD) has similar regulations under Rule 3010 & 3110. Seefor example the SEC web site http://www.sec.gov for further detail.

Another example of an industry where long-term data retention isimportant is the healthcare industry. Regulations require hospitals toretain medical records for a patient's life plus 2 years under HIPAA(Health Insurance Portability and Accountability Act). See for example,the web site http://www.cms.hhs.gov/hipaa/ for further detail.

There are several key issues for long term data preservation, such asfrequency of backups, the storage media, location of the data vault, andso on. One of the most important considerations is faithful datarecovery after many years of storage; i.e., providing exactly the samedata, as it was originally saved, to users after a long period of timehas passed. Generally, users preserve (or archive) data using lower coststorage systems than were used for production data. Examples of lowercost storage systems include tape libraries, optical disk libraries, andATA-based disk storage systems. Compare those systems with typicalhigher performance, higher reliability production data storage systemssuch as a RAID system using FC/SCSI based disks. Since archive storagesystems are lower cost, their reliability is likewise lower than for aproduction system. Therefore, data loss can occur after a long period oftime.

A conventional technique for increasing the reliability andreproducibility of long term data is to use a checksum. Each file is“analyzed” to determine a checksum that is associated with the file. Forexample, each byte (or bytes) of data in the file can be summed toproduce a total called the checksum. The checksum is saved along withthe file. Later, the file can be validated by repeating the checksumcomputation and comparing it with the stored checksum to determine ifthe file has been corrupted over time. Other similar techniques havealso been used, e.g., hash codes. While these methods can detect if thefile has been corrupted, they cannot undo the corruption.

Another conventional technique is to create one or more replicas of thefile and save the file and its replicas on different storage devices.For example, a PCT publication International Publication No. WO 99/38093discloses a method of content addressable information encapsulation,representation, and transfer. As understood, hash values are generatedand used as the file descriptor and the file is replicated in severalstorage resources. The hash value is used to access these replicasuniquely. Since the replica(s) exists in other storage system(s), thefile is recoverable even if the original one is detected to have beencorrupted by using the hash value. However, the method has problems thatthe replicas require extra capacity in the storage systems. As a result,the cost of this solution is relatively expensive.

A need exists for reliable long term data retention. It is desirable toachieve this in a low-cost implementation.

SUMMARY OF THE INVENTION

In accordance with the present invention, one or partitions of an inputfile to be stored on a storage system are identified. A determination ismade for each such input partition whether a sufficient number ofidentical partitions exist in the storage system. One or more replicascan be created to increase the number of identical partitions as needed.Conversely, partitions of a stored file can be read out; e.g., inresponse to a user request to access the file, or during a periodiccheck of stored files. Such readout partitions can be validated. Anydetected corrupt readout partitions can be replaced with a validatedreplacement partition.

BRIEF DESCRIPTION OF THE DRAWINGS

Aspects, advantages and novel features of the present invention willbecome apparent from the following description of the inventionpresented in conjunction with the accompanying drawings, wherein:

FIG. 1 is a generalized block diagram showing an illustrative embodimentof a storage system according to the present invention;

FIG. 2 is an illustration of file processing during a file writeoperation according to the present invention;

FIG. 3 is a high level flow chart highlighting aspects of the presentinvention in a file writing operation;

FIGS. 4 and 4A are illustrative examples of the metadata shown in FIG.1;

FIGS. 5 and 5A are illustrative examples of the partition identityinformation shown in FIG. 1;

FIG. 6 is an illustration of file processing during a file readoperation according to the present invention;

FIG. 7 is a high level flow chart highlighting aspects of the presentinvention in a file reading operation;

FIG. 8 is a generalized block diagram showing an alternative embodimentof the present invention; and

FIG. 9 is a generalized block diagram showing yet another alternativeembodiment of the present invention.

DESCRIPTION OF THE SPECIFIC EMBODIMENTS

FIG. 1 shows a generalized block diagram of an illustrative embodimentaccording to one aspect of the invention. The figure shows a file serverarchitecture for manipulating file-structured data objects, though itwill be appreciated that the invention can be embodied in other storagearchitectures such as storage area networks (SAN) and Object-basedStorage Devices (OSD), and can operate on data objects other than dataorganized as files.

In the illustrative embodiment of FIG. 1, a file server system 1provides file services for one or more clients 50, 51. The file servercan be configured for data communication with one or more storagesystems 70, 71, 72 (also subsystems) in order to access files containedin the one or more file systems supported by the storage systems.Depending on the context, it may be convenient to use the term “storagesystem” as referring to the individual storage systems 70, 71, 72, or touse the term “storage system” to refer to the collection of storagesystems as a single a storage system (or storage subsystem). It can beappreciated that additional file servers can be provided for loadbalancing, to increase redundancy and reliability, and so on. Typically,the storage systems are read/write-able. It can be appreciated that fora particular application of the present invention it might appropriateto use write-once storage media. It can be appreciated that certainoperations discussed below may not be convenient for write-once storagemedia.

The client systems 50, 51 communicate with the file server 1 oversuitable communication links 6. For example, the communication links canbe over a TCP/IP based communication network, such as a local areanetwork (LAN) or a wide area network (WAN). Communication between thefile server 1 and the storage systems 70, 71, 72 can be provided overcommunication links 7 suitable for the architecture being used. Forexample, if the storage systems are SAN-based, then a fibre channelprotocol (FCP) is appropriate. If a network attached storage (NAS)architecture is used, then a TCP/IP based protocol is appropriate. Asanother example, the file server 1 and the storage systems 70, 71, 72can be configured as a single system, in which case the communicationlink 7 can be InfiniBand, PCI, or proprietary protocols. For thepurposes of discussion, a file server architecture is assumed in whichthe interface between the file server 1 and the storage systems is afile interface where the storage systems 70, 71, 72 store data in unitsof files.

A client that performs file write operations is referred to as an “entryclient.” The entry client (e.g., client 50) uses a file writerfunctionality 55 for communicating with the file server 1 to perform awrite operation. In the context of the present invention anymodification to the content of a file is considered “write” operation,including file creation, updates to the file, and file deletion. In aparticular embodiment of the invention, files can be characterized withthe property of “fixity”, which means that the files are written onceand read frequently. Such files are also called “reference information”as well.

A client that performs file read operations is referred to as a “viewclient.” The view client (e.g., client 51) uses a file readerfunctionality 56 to communicate with the file server 1 to access a fileso that its contents can be displayed or otherwise presented to theclient. Typically, any client can possess the ability to write and readfiles via the file server, and is thus can be an entry client or a viewclient, depending on the file operation being performed.

The file server 1 comprises hardware components typically found in afile server system. For example, it is understood that the file servercomprises a computing or other suitable data processing component, asuitable memory component, and is likely to include some form of massstorage (e.g., local hard disk drive). A software component isunderstood to include an operating system (OS) and other supportprograms to control the computing component to communicate with clientsand to communicate with the storage systems 70, 71, 72. It is understoodthat some file system or file systems are defined on the storagesystems, and that the file server contains suitable hardware andsoftware components to access the file systems to provide file storageservices.

In accordance with the embodiment shown in FIG. 1, the file server 1further includes a file input program component 10. The file inputprogram component shown in the figure represents the collection ofsoftware which performs file write operations in accordance with thepresent invention. The file server includes a file output programcomponent 20. The file output program component shown in the figurerepresents the collection of software which performs file accessoperations in accordance with the present invention. The file serverfurther includes metadata 30 and partition identity information 40 whichcan be stored in a suitable mass storage device (e.g., a RAID device).As will become clear, these tables are components in an important aspectof the present invention. Therefore, it might be desirable that thesetables be backed up to other storage devices or replicated to otherstandby system. For example, reliable storage for the tables can beprovided by storing them in a RAID device. The metadata 30 and thepartition identity information 40 are accessible by the softwarecomponents comprising the file input program and the file outputprogram.

One function of the file input program component 10 is to communicatewith an entry client 50 to receive a data stream which constitutes thedata corresponding to a file write request. The file input programcommunicates with the storage system to store the data which constitutesthe file. Additional processes performed by the file input programcomponent is discussed below. In accordance with the particularembodiment of the invention shown in FIG. 1, the file input program alsoupdates the metadata 30 and the partition identity information 40 asneeded.

The file input program component 10 comprises a file partitioning module11. As will be explained, this module identifies the partitions(referred to as input partitions) which constitute the file. A partitionhashing module 12 performs a hash operation. A partition identitytesting module 13 identifies identical partitions. The partitionidentity testing module includes a replication module 14.

One function of the file output program component 20 is to communicationwith a view client 51 to access a requested file from the storage systemand to provide the data to the view client. As will be explained, thisincludes accessing the metadata 30 and may include accessing thepartition identity information 40.

The file output program component 20 comprises a file partitioningmodule 21. This module performs the same function as the filepartitioning module 11, for the file being read. Partitions identifiedin the file being read out are referred to as “readout partitions.” Apartition validating module 22 validates each readout partition whichconstitutes the file being read out. A partition correcting module 23corrects a corrupted readout partition. The partition correcting moduleincludes a partition identity searching module 24.

Referring now to FIGS. 2 and 3, processing a file write operation inaccordance with the particular embodiment of the invention as shown inFIG. 1 will be explained. As discussed above, an entry client 50communicates a request to the file server 1 to perform a file writeoperation. As part of servicing the request, the file is written to thestorage system. In accordance with the present invention, the followingadditional operations are performed on the file. FIG. 3 is a high levelflow chart highlighting the processing which occurs in the file inputprogram 10 of FIG. 1.

In accordance with the invention, a file to written (either for thefirst time in the case of a new file, or as the result of modifying anexisting file) to the storage system is partitioned into one or morepartitions. For each constituent partition, an identical partition iseither located in the storage system, or if none are found a duplicatepartition is created (replica). This is repeated for each constituentpartition, thus ensuring that copy of each partition of a file is to befound somewhere in the storage system. It is appreciated that a inputfile can be smaller that the partition size, and so in that case thefile comprises a single partition. In an alternate embodiment to bediscussed below, each file can be considered a partition, and again afile can comprise a single partition.

Thus, in a particular embodiment of the invention, the constituentpartitions of a file are identified in a step 300, FIG. 3. FIG. 2schematically illustrates the process. The entry client 50 provides afile 100. The partitioning step 300 results in the identification of aplurality of partitions, referred to as input partitions 101–105. Apartition can be defined a fixed-sized blocks of data which comprise thefile. Thus, every N bytes (or bits, or whatever unit is convenient) ofthe file could constitute a partition. The file would be logicallydivided into some number of partitions, each having N bytes (“partitionsize”). The last partition of a file may be smaller than the partitionsize. However, for convenience, it will still be referred to as apartition.

The partition size can be a predetermined size. It is possible toprovide for an administrator to vary the partition size from time totime. The partition size can be programmatically varied in a periodicand automated fashion based on factors such as available storagecapacity, and the so on. Partition size might be determined based onsome aspect of the file; e.g., file type, file size, which specificstorage system 70, 71, 72 the file is located, and so on. For example,all bitmap image files might have a partition size of 1K bits, whiletext files are partitioned into 512 byte blocks.

A loop 310 is performed to process each input partition 101–105 of thefile 100. In accordance with the present invention, each input partitionis identified in terms of its content. In the particular embodimentshown, the content of a partition can be uniquely identified by apartition ID comprising a hash code and a group ID. The hash code willbe discussed now and the notion of a “group ID” will be introducedbelow. The hash code (hash value) is determined by applying some or allof the content of the input partition to a hash function, in a step 320.The hash function can be a one way algorithm; e.g., MD5, SHA-1, and ingeneral can be any suitable algorithm. Thus, for example, FIG. 2 showsthat input partition 101 hashes out to a value of “15”, input partition102 hashes out to a value of “11”, input partition 103 hashes out to“13”, input partition 104 hashes out to “20”, and input partition 10Shashes out to “40”. It can be appreciated that other coding algorithmscan be used instead of hash coding. In addition, it may be desirable toemploy different encoding techniques for different content; e.g., textfiles versus binary files versus bitmap files, and so on.

Returning to the processing of FIG. 3, after a hash value is determinedfor the input partition being processed, the hash value is associatedwith the file 100 as metadata, in a step 330. Referring for a moment toFIG. 4, an illustrative example of metadata 30 is shown. Conventionalfile systems typically store metadata for a file. Metadata refers tofile attributes, location, and other information associated with thefile; i.e., information that is separate from the content of the file.The metadata 30 shown in FIG. 4 is a logical representation of theinformation according to an embodiment of the present invention thateach file might have, presented in a tabular format. The metadata mightinclude a file ID 700, location information 710, and other information720 (e.g., access information, file size, etc.). The metadata associatedwith a file includes a plurality of hash values. There is a hash valuefor each partition identified in the file. For example, the file 100shown in FIG. 2 is shown in the logical representation of FIG. 4 asentry 752, with the hash values for each of the identified partitions.

As mentioned above, the partitioning step 300 can produce differentsized partitions. The metadata 30 can include size information 740.Thus, for example, file entry 751 was partitioned at a size of 512 bytes(or whatever units are convenient).

Continuing with FIG. 3, the next step is to identify whether there is apartition somewhere in the storage system that is identical to the inputpartition being processed. In accordance with this embodiment of thepresent invention, this involves considering each of the partitionswhich constitute each file in the storage system and making adetermination if the partition is the same as the input partition.

Two partitions are “identical” if the content of one partition isbit-by-bit identical to the content of the other partition. Data in apartition (referred to as “partition data”) from one file can beidentical to the partition data from another file. In fact, it ispossible for two or more different partitions from the same file to beidentical (have the same data); e.g., a bitmap file may have a long runof zeroes if the image contains a large white area (or dark area) andthus two or more partitions of such a file might consist only of zeroes.In the particular embodiment shown in FIG. 1, this aspect of theinvention involves accessing the partition identity information 40.

Referring for a moment to FIG. 5, an example of the partition identityinformation 40 is shown. The partition identity information identifiesall the identical partitions and the files containing those partitions.First, consider the partition identifier. This uniquely identifies thecontent of a partition. As mentioned above, the partition identifiercomprises a hash code (step 310) and a group ID. Depending on theparticular implementation specifics, it is likely the hash function willnot guarantee a code that uniquely identifies the content of eachpartition. For example, if a partition size is 256 bytes, and the hashcode is 8 bytes, it is clear that the 8 bytes of the hash code isinsufficient to represent all the possible combinations of 256-bytepartitions. Consequently, it is possible that two partitions which havecontent different from each other will hash out to the same hash value.To further distinguish among those partitions which have the same hashvalue, the “group ID” can be used. Thus, as will be explained below,those partitions which have identical content (i.e., are identical) willbe identified by the same hash code value and by the same group IDvalue. How the group ID is determined will be explained below.

The partition identity information 40 shown in FIG. 5 is logicallyrepresented in tabular form for convenience. Partition identityinformation is provided for each partition of each file stored in thestorage system. Each partition is associated with a partition ID whichcomprises a hash value 800 and a group ID 810. Each partition is furtherassociated with a file ID 820 which identifies the file that containsthe partition. This file ID relates back to the file ID 700 shown inFIG. 4. Each partition is further associated with a partition number830. The partition number is an ordinal number which indicates theposition of that partition in the file relative to the other partitionscomprising the file. Thus, for example, partition entry 851 belongs to afile identified by a file ID of “1000”. This particular partition has ahash value of 13 (and a group ID of 1), and is the 2^(nd) partition inthe file. The grayed out entry 854 represents the addition of a newpartition to the partition information, which will now be discussed.

Returning to the process steps highlighted in FIG. 3, the step ofidentifying whether a partition exists that is identical to the inputpartition being processed in the loop 310 includes retrieving entriesfrom the partition identity information 40 that have the same hashvalue, in a step 340. Thus, suppose the third partition 103 of the fileshown in FIG. 2 is the input partition. This partition hashes out to ahash value of “13”. The partition information table is accessed toidentify other partitions (if any) that have the same hash value. Inthis case, the partitions identified by 851–853 are the candidatepartitions that will be considered in the subsequent steps.

In a loop 350, each candidate partition is subject to the following:

-   -   step 360: The content of candidate partition is accessed. Thus,        for partition 851, the file identified as “1000” would be        accessed; the location can be determined by accessing the        metadata 30 for file “1000”. The content of the 2^(nd) partition        of file “1000” would be read out.    -   step 370: Hash the accessed candidate partition. Thus, the        2^(nd) partition of file “1000” would be hashed.    -   step 380: Compare with the hash value of the candidate with the        hash value of the input partition being processed; they should        be the same. However, if they are different, then it can be        concluded that the candidate partition (in this case, the 2^(nd)        partition of file “1000”) had become corrupted. In accordance        with this particular embodiment of the invention, nothing more        is done for this partition; it is skipped over and processing        proceeds to the top of the loop 350 to process the next        candidate partition(s), in this case partitions 852 and 853.        Alternatively, additional information can be provided in the        partition identity information 40 to indicate that this        candidate partition had been determined to be corrupted. Still        another alternative is to mark this candidate partition for a        subsequent correction step. Yet another alternative is to        attempt to make a correction upon discovering the error. An        example of a correction step is discussed below in connection        with FIG. 7.    -   step 390: If at step 380, the hash values match, then an        identity test is performed. In accordance with this particular        embodiment, the test comprises comparing the content of the        input partition being processed with the accessed candidate        partition to determine if the two partitions are bit-for-bit        identical; i.e. identical partitions.    -   step 400: If the candidate partition and the input partition        being processed are determined to be identical, then the        partition identity information 40 is updated (in a step 410) to        include the input partition. The update information includes the        hash value of the input partition, the group ID of the matching        candidate partition, the file ID of the file to which the input        partition belongs, and the relative position of the input        partition. In this case, the input partition is determined to be        identical, and so a new entry 854 is added to the partition        identity information 40. The hash value is “13” and the group ID        is that of the matching candidate partition, namely, “1”. The        file ID and partition number are also recorded; in this case,        the input partition is from a file identified as “2000” and is        the 3^(rd) partition in the file (see FIG. 2). Continuing with        FIG. 3, processing then proceeds to the top of the loop 310        wherein the next input partition in the input file 100 is        processed.    -   step 400: If the candidate partition and the input partition        being processed are not identical, then processing returns to        the top of the loop 350 to process the next candidate partition        (in this case, the next partition is 852).

If none of the candidate partitions from the loop 350 are identical tothe input partition being processed, then processing proceeds to step420. At this point, it can be concluded that none of the files in thestorage system comprise a partition that is identical to the inputpartition. In step 420, one or more files are created each of whichcomprises the content of the input partition. Such files are referred toas “replicas” or “replica files”. Replica files are distinguished from“user files”, the latter being files created by clients 50, 51, whilethe former are internally created according to the present invention. Bycreating at least one replica of the input partition when an identicalpartition does not exist among the non-replica files stored in thestorage system, a duplicate (i.e., identical) partition of the inputpartition is guaranteed to exist somewhere in the storage system. Inpractice, it is probably preferable to create more than one replica, inorder to assure a certain level of retention reliability. The actualnumber or replicas can be a predetermined value, it can be determined bya system administrator and thus can be changed from time to time, it canbe determined algorithmically in an automated basis, and so on. Inpractice, it is probably preferable to store the replicas throughout thestorage system. Thus, for example, the configuration shown in FIG. 1shows a plurality of storage systems 70, 71, 72. If the file containingthe input partition is stored in storage system 70, for example, then itmight be preferable to store one or more replicas among the storagesystems 71, 72 to reduce the likelihood of lost or corrupted data.

It is worth noting at this point that “partitions” can exist in files asblocks of data (equal to the partition size). The files can beuser-created files, or files created through interaction with a user(user-related files). For example, the component files of a databasesystem are not directly created by a user, but are created to supportthe database which accessed by a user. Files can be replicas ofpartitions which exist in user-related files. Thus, the notion ofreferencing a partition, accessing a partition, or otherwisemanipulating a partition can involve first accessing a file and thenreading out the data that comprises the partition of interest.Alternatively, reference to a partition can simply refer to theinformation contained in the metadata 30 or the partition identityinformation 40.

Continuing with the discussion of FIG. 3, in a step 430, a new group IDis created when a replica is produced. Since it is determined at thispoint that the input partition being processed has no identicalpartition in any of the files stored in the storage system (not countingthe replicas that were just created), a new group ID is created toidentify the new unique partition in the system. In combination with thehash value of the input partition, the resulting new partition IDuniquely identifies the content of the input partition. Allocation of agroup ID can be implemented, for example, by associating a counter witheach hash value. Each counter can be initialized to zero. When a newgroup ID is needed for a given hash value, the counter corresponding tothat hash value can be incremented to produce the new ID. The next timeanother new group ID is needed for that hash value, the counterassociated with that hash value can be incremented again to produce thenext ID. Of course, it can be appreciated from this description thatsome other ID-generating mechanism can be implemented.

In a step 440, the metadata 30 is updated to reflect each of the newlycreated replicas. Similarly, the partition identity information 40 isupdated to include information which identifies the input partitionbeing processed and which identifies each of the newly createdreplica(s). Processing can the repeat at the top of the loop 310 withthe next input partition, until of the input partitions comprising thefile have been processed. In FIG. 2, the loop 310 will be executed fivetimes, once for each partition 101–105.

To illustrate to the foregoing steps, consider FIG. 2. The figureillustrates the situation where partition 102 does not have an identicalpartition in the system. Consequently, the processing of loop 350 willnot yield a partition ID of an identical partition, since none exists.The partition ID for the input partition 102 at this point isrepresented in FIG. 2 as (11, NULL), where “11” is the hash code forpartition 102 and the NULL indicates there are not identical partitions.The replication step 420 is then performed to create one or morereplicas. A new group ID is created per step 430. In the case ofpartition 102, for example, no other partitions in the system hashed outto a value of “11”. Therefore, the group ID value can be “1”, toindicate that it is the first partition having a hash value of “11”. Instep 440, the metadata 30 and the partition identity information 40 areupdated to include the information for partition 102 and for the one ormore replicas.

FIGS. 4A and 5A represent the updates made to the metadata 30 and to thepartition identity information 40. Some additional points are worthnoting. A naming convention can be adopted to identify the replicas. Asthe embodiment in FIG. 4A shows, one such convention can be the use ofspecial file ID. Here, the replica of the partition 102 is identified as“R300”, the “R” indicating that the file is a replica of a partition. Inthis particular embodiment of the invention, each replica contains thepartition data for a single partition. In another embodiment of theinvention, a replica file can store more than one partition, which mayor may not be a less convenient embodiment.

Referring back to step 410, a variation of the disclosed embodiment willnow be discussed. Step 410 is reached if an existing partition that isidentical to the input partition being processed. The partition identityinformation is updated to include information associated with the inputpartition. In addition, the partition identity information can besearched to if the identical partitions comprise partitions from one ormore user files and partitions that are replica files. In such a case,since identical partitions exist in one or more user files, then anyreplicas would be redundant. Therefore, it may be desirable to deleteone or more of the replica files in order to reduce the storage spaceconsumed by the replicas. It is noted that this can be performed by aseparate process.

A further variation of step 410 is to determine from the partitionidentity information the number of identical partitions that match theinput partition being processed, including partitions that belong touser files and partitions that are replicas. In one variation, thenumber of identical partitions can be maintained at some predeterminedvalue so that in step 410 one or more replicas are created (or deleted)to maintain the number of identical partitions at a constant value. If afile is deleted from the storage system, it might be desirable todetermine for each partition in the file to be deleted, whether tocreate one or more replicas. It is noted that still another variation ofstep 410 is that the determination be performed by a separate process.In general, it can be appreciated the steps disclosed the variousfigures can be suitably partitioned into a number of separate processesusing shared memory or other similar data sharing mechanism to passinformation as needed.

In yet another variation, a “buffer zone” can be defined by a firstpredetermined level and a second predetermined level. Thus, if it isdetermined that the number of identical partitions for a given partitionID (hash code, group ID), exceeds the first predetermined level, thenany replicas of that partition can be deleted until the number ofidentical partitions fall below the second predetermined level (or untilall the replicas are deleted, whichever occurs first). This operationcan be performed in step 410, but does not have to be performed in step410. For example, this can be performed by a separate process. The firstand second predetermined levels can be the same level, or differentlevels.

A similar buffer zone can be defined for creating replicas. Thus, if thenumber of identical partitions for a given partition ID falls below athird predetermined level, then enough replicas can be created to sothat the number of identical partitions rises above a fourthpredetermined level. The third and fourth predetermined levels can bethe same level, or different levels.

In accordance with the foregoing embodiment, the processing shown inFIGS. 1 and 3 is initiated when a user requests that a file be stored onthe storage system. It can be appreciated that the process of FIG. 3 canbe performed on files which have already been stored on the storagesystems. Thus, a loop can be provided that, for each file, performs theprocessing of FIG. 3.

Referring now to FIGS. 6 and 7, processing a file read operation inaccordance with the particular embodiment of the invention as shown inFIG. 1 will be explained. As discussed above, a view client 51communicates a request to the file server 1 to perform a file readoperation. As part of servicing the request, the file is accessed fromto the storage system and delivered to the view client. In accordancewith the present invention, the following additional operations areperformed on the file. FIG. 7 is a high level flow chart highlightingthe processing which occurs in the file output program 20 shown inFIG. 1. It will be apparent from the description that follows whichsteps can be performed by the modules which comprise the file outputprogram.

It is noted here that in this particular explanation, the processing ofFIG. 7 is performed in the context of a read operation. One of ordinaryskill can appreciate, however, that the processing of FIG. 7 can beinitiated independently of a read operation. The activity shown in FIG.7 relates to detecting a corrupted file and repairing the detectedcorruption(s). It can be appreciated that such activity can be initiatedother than by a user accessing the file to be read. For example, asystem administrator can issue a command to the system to verify andrepair files stored on the storage system, performed as part of amaintenance operation of stored files which are active or on archivedfiles. An automated process can periodically perform the check, and soon.

In accordance with the invention, each constituent partition of a fileis accessed. For each partition, a determination is made whether thepartition is corrupted. If a partition is corrupted, then an attempt ismade to find an identical partition that is not corrupted. If such anidentical partition is found, then the data in the file of the corruptedpartition is replaced with the data from the un-corrupted identicalpartition. This is repeated for each partition that constitutes thefile.

Referring to FIG. 7 for a particular embodiment of this process, in astep 500, each partition comprising the file is accessed. FIG. 6schematically illustrates this process with the file 200. Thepartitioning step 500 produces a plurality of constituent partitions,201–205, referred to as “readout partitions.” Each readout partition isprocessed as follows in a loop 510:

In a step 520, the hash algorithm used on the partition when the filewas written is applied to the first readout partition 201 to produce ahash value. For example, the example in FIG. 6 shows that readoutpartition 201 hashes out to a hash value (211) of “15”. The readoutpartition 202 has a hash value (212) of “11”, the readout partition 203has a hash value (213) of “14”, and so on.

The hash value produced in step 520 is then compared (in a step 530) tothe value for the readout partition that was produced when the file waswritten. This value is obtained from the metadata 30. Thus, the i^(th)readout partition for a file will have a corresponding hash value in themetadata, namely, it will be the i^(th) value in the metadata for thatfile. If the computed value matches the stored value, then theassumption is made that the readout partition being processed has notbeen corrupted, and is valid. Processing then continues with the loop510 to process the next readout partition 202.

If the computer value does not match the value stored in the metadata30, then a search is performed, in a step 540, for an identicalpartition(s). This includes consulting the partition identificationinformation to identify the list of partitions (“identical partitions”)that are identical to the readout partition being processed. Thus,referring to FIG. 6, an example of a corrupted partition is readoutpartition 203, which has a hash value (213) of “14”; in other words, the3^(rd) partition of file with file ID “2000” has a hash value of “14”.Referring to the metadata 30 shown in FIG. 4A, the 3^(rd) hash value forfile with file ID “2000” is “13”. Since “14” is not the same as “13”, itis determined that the 3^(rd) partition for this file is corrupted.

Thus, for readout partition 203, the partition identificationinformation 40 shown in FIG. 5A is consulted. The partitionidentification information for readout partition 203 is identified basedon the partition's file ID and its ordinal position in the file. Here,the partition is the 3^(rd) partition in the file (file ID “2000”). Thiscorresponds to partition identification information 851 shown in FIG.5A. From this information, it can be determined that the readoutpartition being processed (i.e., partition 203) has a partition ID of(13, 1). Thus, in accordance with step 540, every partition whosepartition ID is (13, 1) will be considered in a loop 550. In the exampleshown in FIG. 5A, file (file ID “1000”) contains a partition (the 2^(nd)partition) that was identified as being identical to the readoutpartition 203.

It is worth noting at this point that the use of replica files ensuresthat each partition has a copy in the system. Whether that copy is validis another matter; however, this aspect of the invention ensures the apartition has at least one identical partition in the system. Forexample, the partition identification information of FIG. 5A shows thatthe only partition that is identical to the second partition in file(file ID “2000”) is the partition stored in the replica file (file ID“R300”).

Continuing, in a step 560, each such “candidate” partition is read outfrom the storage system. This involves accessing the file in which thecandidate partition belongs. The hash algorithm that was initiallyapplied to the candidate partition when its corresponding file waswritten to the storage system is applied to the candidate partition toproduce a hash value, in a step 570. The hash value is compared (in astep 580) against the value stored in the metadata corresponding to thecandidate partition. If a match is not found, then processing continueswith loop 550 to consider the next candidate partition. If no morecandidate partitions are found, then in a step 600 an error conditioncan be reported to a user or logged for review at a later time. Theerror condition indicates that the file has been corrupted.

If a match has been found, then the candidate partition is deemed to bevalidated. In a step 590, the data comprising the readout partitionbeing processed, which is corrupted, is replaced by the data comprisingthe validated candidate partition. This operation is performed in thefile output program 20 (FIG. 1) using the file I/O utilities formodifying files in the supported file system. For example, the partitioncorrecting module 23 can perform this aspect of the invention.Processing then returns to loop 510 to process the next readoutpartition.

The foregoing described embodiment of the present invention is wellsuited for existing storage system architectures. In the foregoingdescribed embodiment, the constituent partitions of a file are notphysically stored, except for the replica files. The partitions of afile are logical partitions. If a partition is 1024 bytes in size, thenthe act of “identifying” the n^(th) partition may simply involve readingblocks of 1024 bytes of data and retaining the n^(th) block. The hashalgorithm can then be applied to that block of data. The next block isaccessed simply by reading the next 1024 bytes of data.

It can be appreciated, however, that a storage system can be configuredto store files in units of partitions. Such a storage architecture mightprovide improved performance over legacy systems by being optimized foraccessing partitions. Depending on the implementation, one of ordinaryskill will realize that modifications are likely needed for theinformation comprising the metadata 830 and the information comprisingthe partition identity information 840. A well known example of such astorage system is object based storage. SNIA (Storage Network IndustryAssociation) has been working for OSD (Object Storage Devices) tostandardize object based storage. In this case, the partition may becalled an “object.”

FIG. 8 is an illustrative example of an embodiment of this aspect of theinvention. A primary difference from the system shown in FIG. 1 is thepresence of storage systems 870, 871, and 872. The storage systems areshown storing information in units of partitions 880, 881, and 882,respectively, as compared to storage systems 70, 71, 72 of FIG. 1 whichstore information in units of files 80, 81, 82. Likewise, partitionreplicas 890, 891, 892 would be different from the replica files 90, 91,92 of FIG. 1.

A file server 801 might have access to additional functionality providedby the storage systems 870, 871, 872. For example, the file server mighthave Object Relocation functionality to relocate or arrange partitionsamong the storage systems based on a predefined user's policy. If theuser wants to increase the redundancy of the whole system, an attemptcan be made to relocate the partitions of a file to different storagesystems.

The components of the file server 801 according to this embodiment ofthe invention might not require the file partitioning module 11 in thefile input program 10 or the file partitioning module 21 in the fileoutput program 20. The reason is that the partitioning function isprovided by the storage systems 870, 871, 872. However, it my bedesirable to partition the files using a partition size that isdifferent than what is provided by the storage systems, in which casethe partitioning modules 11, 21 might be needed. The modules are shownin gray outline to indicate they may or may not be needed components.

Referring to FIG. 9, a discussion of an alternate embodiment of thepresent invention will now follow. The system configuration shown hereis similar to the configuration shown in FIG. 1. Users 50, 51 access afile server 1 a via suitable communication links 6. Storage systems 70a, 71 a, 72 a provide data storage capability for the users. Suitabledata paths 7 provide a data link between the file server and the storagesystems.

The file server 1 a includes a file input program 10 a and a file outputprogram 20 a. The operations performed by the modules which constitutethe file input program and the file output program are similar to theircounterparts shown in FIG. 1. In this particular embodiment of theinvention, there is no partitioning of an input file that is to bewritten to the storage system. Likewise, there is no accessing ofpartitions of a file that is being read out of the storage system.

Instead of multiple constituent partitions of a file, the entire file istreated as a single large partition. Consequently, there is nopartitioning module; however, the other modules comprising the fileinput program 10 a and the file output program 20 a operate in a mannersimilar to the process flows shown in FIGS. 3 and 7. Thus, a filehashing module 12 a applies a hash function to the entire contents of ainput file. Similarly, a file validating module 22 a includes applyingthe hash function to the entire contents of the file. It can beappreciated that the hash function (or any suitable algorithm forgenerating a content-based code) can be applied to a portion of the file(e.g., every other byte). This particular embodiment of the presentinvention simply points out that partitioning of the file is not beingperformed.

For an input file, a file identity module 13 a identifies identicalfiles rather than identical partitions. Replicas are created by a module14 a. Replicas therefore are copies of the file. Thus, the replicas 90a, 91 a, 92 a stored in the storage systems 70 a, 71 a, 72 a are copiesof files. For reading out a file, a correction module 23 a repairs afile by looking (via module 24 a) for a identical validated files. Tocomplete the discussion of FIG. 9, the metadata 30 a requires only asingle hash value for each file, namely, the hash value of the file. Thepartition identity information 40 shown in FIG. 1 is replaced with fileidentity information 40 a, which provides the same kind of informationas the partition identity information absent the partition number 830component.

The present invention is particularly useful in any application areathat requires long term data preservation, which also requires dataprotection for long period automatically. Although the foregoingembodiments of the present invention were described with respect tofiles as the unit of data, it can be appreciated that units of dataother than files can be processed. Information can be stored in manyforms, depending on specific requirements of the user. A sampling ofapplications include:

-   -   Digital Imaging, where the historical value of data is very        important.    -   Email Archiving, where the same message and the same attachment        files are distributed many users, but archive them effectively        with efficient disk spaces.    -   Content/Document Archiving, where version-up are repeated and as        a result the same data parts are contained.    -   Medical Imaging, where data must be accurate for long period.    -   Desktop Archiving, where data on all desktops in an office are        archived, and usually most of the data are the same, since users        use the same environment usually in the office.

1. A method for accessing a storage system comprising: accessing a dataobject, the data object being divisible into one or more partitions, thepartitions comprising data from the data object, the partitions referredto as input partitions; and for each input partition, if there are noother partitions among other data objects in the storage system that areidentical to the input partition, then producing one or more replicas ofthe input partition.
 2. The method of claim 1 wherein the data object isa file.
 3. The method of claim 1 wherein a first partition is associatedwith a partition ID, such that another partition having content that isidentical to content of the first partition, then the other partition isassociated with the same partition ID as that of the first partition,the method further comprising storing partition identificationinformation comprising a plurality of partition IDs, each partition IDbeing associated with one or more partitions, determining a partition IDof the input partition, accessing the partition identity information todetermine if there are any partitions that are identical to the inputpartition based on the partition ID of the input partition.
 4. Themethod of claim 1 wherein the data object is to be stored on the storagesystem, the method further comprising receiving a request to store thedata object on the storage system, receiving data comprising the dataobject, and storing the data object on the storage system.
 5. The methodof claim 1 further comprising, for each input partition, generating acontent-based identifier based on at least some content of the inputpartition and identifying first partitions in the storage system thathave the same content-based identifier, wherein the one or more replicasare produced if none of the first partitions is identical to the inputpartition.
 6. The method of claim 5 wherein the step of generatingincludes applying a hash algorithm to at least a portion of the contentof the input partition.
 7. The method of claim 1 wherein the data objectis a file.
 8. A method for accessing a storage system comprising:receiving data for a first file, to be stored in the storage system;providing partition data from the first file which constitutes a firstpartition of the first file; if a number of second partitions in thestorage system is less than a first predetermined value, then producinga number of replicas of the first partition sufficient to increase thenumber of second partitions to a second predetermined value, whereineach second partition comprises data belonging to a file in the storagesystem and is identical to the partition data; and if the number ofsecond partitions is greater than a third predetermined value and ifthere are one or more replicas of the first partition, then deleting oneor more of the replicas, wherein the number of second partitions isreduced; and repeating for additional partition data comprising thefirst file.
 9. The method of claim 8 further comprising receiving arequest to store the first file, receiving data comprising the firstfile, and storing the first file on the storage system.
 10. The methodof claim 8 wherein partitions each is identified by a content-based codeand a group ID, wherein if data one partition is different from data ofanother partition and both partitions have the same content-based code,then each partition has a different group ID, whereby partitions thatcontain identical data are identified by the same content-based code andgroup ID.
 11. The method of claim 10 further comprising storingpartition identity information comprising a content-based code and agroup ID that correspond to each partition on the storage system,wherein the first partition is associated with a first content-basedcode value and a first group ID value, wherein the number of secondpartitions can be determined by consulting the partition identityinformation and counting the number of partitions whose correspondingcontent-based code is equal to the first content-based code value andwhose corresponding group ID is equal to the first group ID value. 12.The method of claim 10 further comprising storing partition identityinformation comprising a hash code and a group ID that correspond toeach partition on the storage system, wherein the first partition isassociated with a first hash code value and a first group ID value,wherein the step of producing a number of replicas includes addinginformation which identifies each replica to the partition identityinformation, including the first hash code value and the first group IDvalue.
 13. The method of claim 8 wherein the content-based code is ahash code produced by applying a hash algorithm to content of apartition.
 14. The method of claim 8 wherein the first predeterminedvalue is less than the second predetermined value.
 15. The method ofclaim 8 wherein the first predetermined value is equal to the secondpredetermined value.
 16. The method of claim 8 wherein the step ofdeleting one or more replicas further includes deleting one or morereplicas until all the replicas are deleted or until the number ofsecond partitions is less than a fourth predetermined value.
 17. Themethod of claim 16 wherein the third predetermined value is greater thanthe fourth predetermined value.
 18. A method for accessing a storagesystem comprising: receiving a request to store a file; storing the fileon the storage system; identifying one or more partitions whichcollectively constitute the file, the partitions referred to as inputpartitions; storing partition information that is associated with thefile, wherein the partition information associates the file with each ofits input partitions; and for each input partition, if there are noidentical partitions, then if the number of replicas of the inputpartition is less than a threshold value, then producing at least onereplica of the input partition and storing the replica on the storagesystem, wherein an identical partition is a partition, other than theinput partition, of a file that is stored in the storage system whosecontent is identical to content of the input partition.
 19. The methodof claim 18 wherein for a first input partition, if there is at leastone other file that comprises a partition that is identical to the firstinput partition, then deleting a replica of the first input partition ifthe replica exists.
 20. The method of claim 18 wherein for an inputpartition, the number of identical partitions plus the number ofreplicas is equal to a predetermined value, the threshold value beingthe difference between the predetermined value and the number ofidentical partitions.
 21. The method of claim 18 wherein the partitioninformation associated with the file comprises a partition ID for eachinput partition, wherein the partition ID comprises a hash code and agroup ID, wherein the hash code is determined by applying a hashfunction to contents of a partition, wherein the group ID identifies apartition whose content is unique among partitions which have the samehash code.
 22. The method of claim 18 further comprising storinginformation that identifies one or partition groups, a partition groupcomprising one or more partitions identified from among one or morefiles which contain identical content, a partition group furthercomprising one or more replicas of a partition in the partition group.23. The method of claim 18 wherein the threshold value is one.
 24. Themethod of claim 18 wherein the threshold value is a number greater thanone.
 25. The method of claim 18 wherein each partition is the same sizeas other partitions.
 26. The method of claim 18 wherein identifying oneor more partitions includes determining a partition size by which thepartitions of the file are identified.
 27. The method of claim 18wherein a partition size of partitions of a file can be different fordifferent files.
 28. A data storage system comprising: a storagecomponent; and a data processing component in data communication withthe storage component, the data processing component for receivingaccess requests from users, the access requests for accessing data thatis stored in the storage component or for storing data to the storagecomponent, the data processing component configured to perform themethod steps of: accessing a first partition of a file, the firstpartition comprising a first portion of data that constitutes the file;if the first partition does not have a corresponding identical partitionin the storage component, then creating at least one replica; andrepeating for a second partition of the file, the second partitioncomprising a second portion of the data.
 29. The system of claim 28wherein the data processing component is further configured to performthe method steps of: receiving a request to store data to the storagesystem, the request including the data that constitutes the file; andaccessing the storage component to store the data.
 30. The system ofclaim 28 wherein a partition is identified with a partition ID, thepartition ID being based on content of the partition, wherein partitionswhich contain identical content have the same partition ID, the storagesystem further comprising information which is stored in the storagecomponent, each partition having its associated partition identityinformation, the partition identity information comprising a partitionID its associated partition, wherein partitions that are identical havethe same partition ID, whereby identical partitions can be identified byconsulting the partition identity information.
 31. The system of claim30 wherein the partition ID comprises a content-based code and a groupID, wherein the content-based code is determined from the content of apartition, wherein if one partition and another partition have the samecontent-based code but have different content, then the one partition isassociated with a first group ID and the other partition is associatedwith a second group ID different from the first group ID, wherein if theone partition and the other partition have the same content-based codeand have identical content, then the one partition and the otherpartition both are associated with the same group ID.
 32. The system ofclaim 31 wherein the content-based code is a hash code, wherein the hashcode is generated by applying a hash function to the content of apartition.
 33. A data storage system comprising: first means forproducing a partition ID for each partition comprising a file stored inthe storage system, a partition comprising data from the file, the firstmeans producing a first partition ID for a first partition of a firstfile; second means for identifying one or more identical partitionsamong other files in the storage system based on a first partition ID;and third means for creating a replica of the first partition inresponse to the second means making a determination that there are noidentical partitions, wherein the first means, the second means, and thethird means operate on every partition comprising the first file. 34.The system of claim 33 wherein a partition ID comprises a hash code anda group ID, wherein the first means comprising hash means for producinga hash code based on data comprising a partition and group ID means fordetermining a group ID.